[Klaud Cold] Add H200 DeepSeek V4.1 Flash AgentX / 新增 H200 配方 - #2963
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
3 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @pytest.mark.parametrize("occupied", [False, True]) | ||
| def test_select_available_server_port_avoids_an_existing_listener(occupied: bool) -> None: | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: | ||
| listener.bind(("0.0.0.0", 0)) |
| else: | ||
| assert selected == preferred | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server: | ||
| server.bind(("0.0.0.0", selected)) |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I also checked the new select_available_server_port() helper in benchmarks/benchmark_lib.sh for the same class of check-then-act issue — it does have a bind-then-release TOCTOU window before the caller's vllm serve rebinds the port, but this is a pre-existing pattern shared byte-identically with sibling PR #2960 and the window is negligible for single-node exclusive benchmark launches, so I'm not flagging it as a separate blocker.
Extended reasoning...
A confirmed, CI-blocking bug was already reported inline: perf-changelog.yaml:7080 sets pr-link: TBD, which utils/validate_perf_changelog.py's PR_LINK_PLACEHOLDERS set (only XXX or pull/XXX) does not accept, so validate_added_pr_link will raise on any CI run until the author swaps in the real PR link or the XXX placeholder. I independently verified this by reading the validator source, confirming the claim is accurate. Given a confirmed finding exists, approval is not appropriate here, and the inline comment already signals that human review is needed, so my body is limited to the narrow ruled-out exception: I additionally examined the new select_available_server_port() function added in benchmarks/benchmark_lib.sh for a similar race-condition class of bug (binding a socket, closing it, then relying on the freed port remaining free for vllm serve). This is a real TOCTOU window, but it is inherent to this common "find a free port" pattern, the risk window is small in the context of exclusive single-node benchmark launches, and per the PR description this code is byte-identical to sibling PR #2960's already-reviewed helper, so I did not treat it as a new blocking issue for this PR.
| description: | ||
| - "Add Hopper AgentX arms for DeepSeek-V4.1-Flash at TP8 with native five-token DSpark and adaptive real verification" | ||
| - "Engram tables offload to host DRAM through UVA while the KV cache stays GPU-resident; H200 runs concurrency 1-128 and H100 stops at 16 on its smaller KV headroom" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The new changelog entry uses pr-link: TBD, but utils/validate_perf_changelog.py only accepts the canonical .../pull/<N> link or the literal placeholder XXX (or pull/XXX) for PR-time entries — TBD matches neither, so validate_added_pr_link raises ChangelogValidationError and the changelog gate (run in run-sweep.yml) fails on this PR as-is. Fix: change the placeholder to XXX (or the actual PR number) to match PR_LINK_PLACEHOLDERS.
Extended reasoning...
utils/validate_perf_changelog.py defines PR_LINK_PLACEHOLDERS = {"XXX", ".../pull/XXX"} and validate_added_pr_link raises when link not in PR_LINK_PLACEHOLDERS and link != expected. The new entry at perf-changelog.yaml:7080 sets pr-link: TBD, which is not in that set, so any CI run of validate_perf_changelog.py (wired into run-sweep.yml) on this PR fails the changelog gate, blocking merge/sweep until the author swaps TBD for XXX or the real PR number.
Verification: normal (with a corrected enforcement point). The defect is real: perf-changelog.yaml:7080 uses pr-link: TBD, which is neither the canonical .../pull/<N> link nor a member of PR_LINK_PLACEHOLDERS = {"XXX", ".../pull/XXX"} (utils/validate_perf_changelog.py:27-30). validate_added_pr_link (line 159) raises ChangelogValidationError for any link that is not a placeholder and not the…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34463999964 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34467029236 |
Oseltamivir
changed the title
Add the Hopper arms of the DeepSeek-V4.1-Flash DSpark AgentX recipe at TP8, sharing the text-only vLLM serving script with the Blackwell arms. Engram tables offload to host DRAM through UVA while the KV cache stays GPU-resident, which is what brings the 511 GB checkpoint under 8x80 GB on H100. The H100 launcher now prefers the framework-tagged benchmark script name, as the H200 launchers already did. 新增 DeepSeek-V4.1-Flash DSpark AgentX 配方的 Hopper 分支，采用 TP8，并与 Blackwell 分支共用纯文本 vLLM 服务脚本。Engram 表通过 UVA 卸载到主机 DRAM， KV cache 仍驻留 GPU，这使 511 GB 的检查点得以放入 8x80 GB 的 H100。H100 launcher 现在优先解析带 framework 的基准脚本名称，与 H200 launcher 保持一致。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
补充 perf-changelog 的 pr-link。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DeepSeek-V4.1-Flash serves 1M context and must replay semianalysis_cc_traces_weka_062126, not the 256k-capped variant. The recipe inherits that only because resolve_trace_source's dsv4* case arm also matches the dsv41flash prefix, which is invisible at the call site, so pin it with a test and document the dependency. DeepSeek-V4.1-Flash 服务 1M 上下文，必须回放 semianalysis_cc_traces_weka_062126，而非 256k 截断的变体。配方获得该语料仅仅 是因为 resolve_trace_source 的 dsv4* 分支同时匹配了 dsv41flash 前缀，而这一 依赖在调用处并不可见，因此用测试固定并补充文档说明。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first sweep OOMed on H100 at concurrency 1: at 1M context the sparse attention indexer allocates a [max-num-batched-tokens, max-model-len] logits buffer in fp8_fp4_paged_mqa_logits, exactly 16 GiB at the default 8192 batched tokens. That is a fixed startup cost paid during memory profiling, independent of concurrency, so an 80 GB card cannot host it alongside the weights even though the Engram offload leaves them fitting. H200 eval passed on the same commit, so drop the H100 arm here and track it separately. 首次 sweep 在 H100 并发 1 时 OOM：在 1M 上下文下，稀疏注意力 indexer 会在 fp8_fp4_paged_mqa_logits 中分配 [max-num-batched-tokens, max-model-len] 的 logits 缓冲区，在默认 8192 batched tokens 下恰好为 16 GiB。这是显存 profiling 阶段固定支付的启动开销，与并发无关，因此尽管 Engram 卸载后权重放得下，80 GB 的卡仍无法同时容纳。同一 commit 上 H200 的 eval 已通过，故在此移除 H100 分支 并另行跟踪。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DSv4.1 Flash 吞吐测试使用 thinking 开启、五个草稿 token 对应的黄金合成 AL 3.51，关闭自适应验证；准确率评测保留真实验证。同步中英文文档并追加性能变更记录。
Follow-up to #2963, which dropped this arm after it OOMed at concurrency 1. The blocker is the sparse attention indexer's [max-num-batched-tokens, max-model-len] buffer, 16 GiB at 1M context with the shared script's 8192 batched tokens. Give H100 its own script that caps batched tokens at 4096, sets max-num-seqs from the trajectory concurrency instead of vLLM's default 1024, and enables expandable_segments. Also teach the h100 launcher to resolve framework-tagged script names, which it never could. 承接 #2963（该 PR 在此分支于并发 1 OOM 后将其移除）。瓶颈是稀疏注意力 indexer 的 [max-num-batched-tokens, max-model-len] 缓冲区，在 1M 上下文下按共享脚本的 8192 batched tokens 计为 16 GiB。为 H100 提供独立脚本：将 batched tokens 限制为 4096、 依轨迹并发设置 max-num-seqs（而非 vLLM 默认的 1024）、启用 expandable_segments。 同时让 h100 launcher 能够解析带 framework 的脚本名，此前它完全无法解析。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2defaf7 to
5c5a5a6
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34485011650 |
perf-changelog.yaml is append-only, and the rebase onto main dropped PR #2969's dsv4-fp4-gb300-dynamo-sglang-agentic-agg entry, which fails check-changelog. Rebuild the file as main's content plus this branch's own appended entries. perf-changelog.yaml 为只可追加文件，而向 main 的 rebase 丢掉了 PR #2969 的 dsv4-fp4-gb300-dynamo-sglang-agentic-agg 条目，导致 check-changelog 失败。 将该文件重建为 main 的内容加上本分支自身追加的条目。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
同步最新 main，保留性能变更日志历史字节，并在末尾保留本 PR 的变更记录。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f943039. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34504985992 |
|
/reuse-sweep-run |
Resolve conflicts with the H200 AgentX arm (#2963): keep both master-config entries and both docs sections, take main's Chinese wording in MODELS_zh.md, keep the overridable engine-readiness default in the shared script, and append this branch's changelog entries at the tail. 将 main 合并入 GB300 DSv4.1 Flash AgentX 分支，解决与 H200 分支（#2963）的冲突： 保留两个 master 配置条目与两段文档，MODELS_zh.md 采用 main 的中文表述，共享脚本 保留可覆盖的引擎就绪默认值，并将本分支的 changelog 条目追加到文件末尾。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolve conflicts with the merged H200 (#2963) and GB300 (#2961) arms: keep all three master-config entries, take main's docs and MODELS_zh.md wording, keep the overridable engine-readiness default in the shared script, and append this branch's changelog entries at the tail. 将 main 合并入 B200 DSv4.1 Flash AgentX 分支，解决与已合入的 H200（#2963）和 GB300（#2961）分支的冲突：保留三个 master 配置条目，文档与 MODELS_zh.md 采用 main 的表述，共享脚本保留可覆盖的引擎就绪默认值，并将本分支的 changelog 条目追加到末尾。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolve conflicts with the merged H200 (#2963), GB300 (#2961), and B200 (#2960) arms: keep all four master-config entries, take main's docs and MODELS_zh.md wording, keep the overridable engine-readiness default in the shared script, and append this branch's changelog entries at the tail. 将 main 合并入 B300 DSv4.1 Flash AgentX 分支，解决与已合入的 H200（#2963）、 GB300（#2961）和 B200（#2960）分支的冲突：保留四个 master 配置条目，文档与 MODELS_zh.md 采用 main 的表述，共享脚本保留可覆盖的引擎就绪默认值，并将本分支的 changelog 条目追加到末尾。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolve conflicts with the merged H200 (#2963), GB300 (#2961), and B200 (#2960) arms: keep all four master-config entries, take main's docs and MODELS_zh.md wording, keep the overridable engine-readiness default in the shared script, and append this branch's changelog entries at the tail. 将 main 合并入 GB200 DSv4.1 Flash AgentX 分支，解决与已合入的 H200（#2963）、 GB300（#2961）和 B200（#2960）分支的冲突：保留四个 master 配置条目，文档与 MODELS_zh.md 采用 main 的表述，共享脚本保留可覆盖的引擎就绪默认值，并将本分支的 changelog 条目追加到末尾。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 participants
Add H200 TP8 DSv4.1 Flash AgentX at concurrency 1–128, using the full
semianalysis_cc_traces_weka_062126corpus. Five-token DSpark throughput uses golden AL 3.51 (thinking on), with adaptive verification disabled; accuracy evals retain real block verification. Engram stays in host DRAM through UVA; KV stays on GPU. Validation: Bash, exact-key matrix, changelog, launcher/eval tests, and both serving-mode command checks pass. GPU validation pending.新增 H200 TP8 DSv4.1 Flash AgentX，并发 1–128，使用完整
semianalysis_cc_traces_weka_062126语料。五 token DSpark 吞吐测试采用 thinking 开启时的黄金 AL 3.51，并关闭自适应验证；准确率 eval 保留真实块验证。Engram 通过 UVA 访问主机 DRAM，KV 保留在 GPU。验证：Bash、精确配置矩阵、changelog、launcher/eval 测试及两种服务模式的命令检查通过；GPU 验证待完成。Note
Medium Risk
New cluster benchmark recipe and Slurm launcher behavior for a model prefix, plus shared port-selection helper used at serve time; GPU validation is still pending.
Overview
Adds DeepSeek-V4.1-Flash (
dsv41flash) to the model matrix and introduces the H200 TP8 AgentX recipedsv41flash-fp4-h200-vllm-agentic-dsparkinnvidia-master.yaml, with perf-changelog and EN/zh configuration docs (including rationale for no H100 arm).The new vLLM agentic launcher serves 1M context with Engram CPU UVA offload, GPU-resident KV, five-token DSpark (throughput pins golden AL 3.51; eval keeps real verification), and replays the uncapped
semianalysis_cc_traces_weka_062126corpus via the existingdsv4*trace default.Infrastructure:
select_available_server_port()inbenchmark_lib.shavoids host-network port collisions; H200 Slurm launcher mounts the repo at/ixfordsv41flashso AgentX runtime dirs stay out of/workspace. Tests cover launcher mounts, trace corpus selection, and port probing.Reviewed by Cursor Bugbot for commit 5de26e3. Bugbot is set up for automated code reviews on this repo. Configure here.